BiPMAP: a toolbox for predicting perceived motion artifacts on modern displays.

Viewers of digital displays often experience motion artifacts (e.g., flicker, judder, edge banding, motion blur, color breakup, depth distortion) when presented with dynamic scenes. We developed an interactive software tool for display designers that predicts how a viewer perceives motion artifacts for a variety of stimulus, display, and viewing parameters: the Binocular Perceived Motion Artifact Predictor (BiPMAP). The tool enables the user to specify numerous stimulus, display, and viewing parameters. It implements a model of human spatiotemporal contrast sensitivity in order to determine which artifacts will be seen by a viewer and which will not. The tool visualizes the perceptual effects of discrete space-time sampling on the display by presenting side by side the expected perception when the stimulus is continuous compared to when the same stimulus is presented with the spatial and temporal parameters of a prototype display.

The Statistics of Eye Movements and Binocular Disparities during VR Gaming: Implications for Headset Design.

The human visual system evolved in environments with statistical regularities. Binocular vision is adapted to these such that depth perception and eye movements are more precise, faster, and performed comfortably in environments consistent with the regularities. We measured the statistics of eye movements and binocular disparities in virtual-reality (VR) - gaming environments and found that they are quite different from those in the natural environment. Fixation distance and direction are more restricted in VR, and fixation distance is farther. The pattern of disparity across the visual field is less regular in VR and does not conform to a prominent property of naturally occurring disparities. From this we predict that double vision is more likely in VR than in the natural environment. We also determined the optimal screen distance to minimize discomfort due to the vergence-accommodation conflict, and the optimal nasal-temporal positioning of head-mounted display (HMD) screens to maximize binocular field of view. Finally, in a user study we investigated how VR content affects comfort and performance. Content that is more consistent with the statistics of the natural world yields less discomfort than content that is not. Furthermore, consistent content yields slightly better performance than inconsistent content.

The visual benefits of correcting longitudinal and transverse chromatic aberration.

We describe a system-the Binocular Varichrome and Accommodation Measurement System-that can be used to measure and correct the eye's longitudinal and transverse chromatic aberration (LCA and TCA) and to perform vision tests with custom corrections. We used the system to investigate how LCA and TCA affect visual performance. Specifically, we studied the effects of LCA and TCA on visual acuity, contrast sensitivity, and chromostereopsis. LCA exhibited inter subject variability but followed expected trends compared with previous reports. TCA at the fovea was variable between individuals but with a tendency for the shift at shorter wavelengths to be more temporalward in the visual field in each eye. We found that TCA was generally greater when LCA was corrected. For visual acuity, we found that a measurable benefit was realized only with both LCA and TCA correction unless the TCA was low. For contrast sensitivity, we found that the best sensitivity to a 10-cycle/degree polychromatic grating was attained when LCA and TCA were corrected. Finally, we found that the primary cause of chromostereopsis is the TCA of the eyes.

Suprathreshold Contrast Perception Is Altered by Long-term Adaptation to Habitual Optical Blur.

Purpose: To investigate whether visual experience with habitual blur alters the neural processing of suprathreshold contrast in emmetropic and highly aberrated eyes. Methods: A large stroke adaptive optics system was used to correct ocular aberrations. Contrast constancy was assessed psychophysically in emmetropic and keratoconic eyes using a contrast matching paradigm. Participants adjusted the contrasts of gratings at various spatial frequencies to match the contrast perception of a reference grating at 4 c/deg. Matching was done both with fully corrected and uncorrected ocular aberrations. Optical correction allowed keratoconus patients to perceive high spatial frequencies that they have not experienced for some time. Results: Emmetropic observers exhibited contrast constancy both with their native aberrations and when their aberrations were corrected. Keratoconus patients exhibited contrast constancy with their uncorrected, native optics but they did not exhibit constancy during adaptive optics correction. Instead. they exhibited striking underconstancy: they required more contrast at high spatial frequencies than the contrast of the 4-c/deg stimulus to make them seem to have the same contrast. Conclusions: The presence of contrast constancy in emmetropes and keratoconus patients viewing with their native optics suggests that they have learned to amplify neural signals to offset the effects of habitual optical aberrations. The fact that underconstancy was observed in keratoconus patients when their optics were corrected suggests that they were unable to learn the appropriate neural amplification because they did not have experience with fine spatial detail. These results show that even adults can learn neural amplification to counteract the effects of their own optical aberrations.

Radial and Tangential Retinal Magnifications as Functions of Visual Field Angle Across Spherical, Oblate, and Prolate Retinal Profiles.

Purpose: To provide a tool for calculating radial and tangential retinal magnifications as functions of field angle and retinal shape and to articulate patterns of magnification across the retina for monocular and binocular combinations of prolate-, oblate-, and spherical-shaped retinas. Methods: Formulae were derived to calculate radial and tangential retinal magnifications (mm/deg) from field angle (degrees), retinal asphericity (unitless conic constant), retinal vertex radius of curvature (mm), and nodal point position (mm). Monocular retinal magnifications were determined for eyes with prolate, spherical, and oblate retinas as functions of field angle. Bilateral differences in magnifications were examined for combinations of those eyes. Results: Retinal shape substantially affects magnification profiles even for eyes with the same axial length. Greatest magnification changes across a retina and between eyes, as well as greatest increase in radial-tangential differences (distortion), occur with prolate retinas. Binocular magnification differences were smallest for oblate retinas. Nodal points anterior to the vertex center of curvature and oblate asphericity both cause field-dependent reductions in magnification relative to the fovea (barrel distortion), whereas nodal points posterior to vertex center of curvature and prolate asphericity cause the opposite (pincushion distortion). Retinal magnification differences due to eye shape are much greater than aniseikonia thresholds and chromatic differences in magnification. A spreadsheet tool implements the magnification calculations. Conclusions: Local retinal magnifications as functions of field angle have substantial effects on objective applications (imaging retinal anatomy) and subjective experiences (aniseikonia) and quantify an ocular property that differs across eye shapes and refractive errors. Translational Relevance: Methods are provided to customize the calculation of radial and tangential magnifications across the retina for individual eyes, which will bolster the multifactorial study of the effects of foveal and peripheral optics across eye shapes and refractive errors.

High-fidelity eye, head, body, and world tracking with a wearable device.

We describe the design and performance of a high-fidelity wearable head-, body-, and eye-tracking system that offers significant improvement over previous such devices. This device's sensors include a binocular eye tracker, an RGB-D scene camera, a high-frame-rate scene camera, and two visual odometry sensors, for a total of ten cameras, which we synchronize and record from with a data rate of over 700 MB/s. The sensors are operated by a mini-PC optimized for fast data collection, and powered by a small battery pack. The device records a subject's eye, head, and body positions, simultaneously with RGB and depth data from the subject's visual environment, measured with high spatial and temporal resolution. The headset weighs only 1.4 kg, and the backpack with batteries 3.9 kg. The device can be comfortably worn by the subject, allowing a high degree of mobility. Together, this system overcomes many limitations of previous such systems, allowing high-fidelity characterization of the dynamics of natural vision.

Binocular vision and the control of foot placement during walking in natural terrain.

Coordination between visual and motor processes is critical for the selection of stable footholds when walking in uneven terrains. While recent work (Matthis et al. in Curr Biol 8(28):1224-1233, 2018) demonstrates a tight link between gaze (visual) and gait (motor), it remains unclear which aspects of visual information play a role in this visuomotor control loop, and how the loss of this information affects that relationship. Here we examine the role of binocular information in the visuomotor control of walking over complex terrain. We recorded eye and body movements while normally-sighted participants walked over terrains of varying difficulty, with intact vision or with vision in one eye blurred to disrupt binocular vision. Gaze strategy was highly sensitive to the complexity of the terrain, with more fixations dedicated to foothold selection as the terrain became more difficult. The primary effect of increased sensory uncertainty due to disrupted binocular vision was a small bias in gaze towards closer footholds, indicating greater pressure on the visuomotor control process. Participants with binocular vision losses due to developmental disorders (i.e., amblyopia, strabismus), who have had the opportunity to develop alternative strategies, also biased their gaze towards closer footholds. Across all participants, we observed a relationship between an individual's typical level of binocular visual function and the degree to which gaze is shifted toward the body. Thus the gaze-gait relationship is sensitive to the level of sensory uncertainty, and deficits in binocular visual function (whether transient or long-standing) have systematic effects on gaze strategy in complex terrains. We conclude that binocular vision provides useful information for locating footholds during locomotion. Furthermore, we have demonstrated that combined eye/body tracking in natural environments can be used to provide a more detailed understanding of the impact of a type of vision loss on the visuomotor control process of walking, a vital everyday task.

Optics and neural adaptation jointly limit human stereovision.

Stereovision is the ability to perceive fine depth variations from small differences in the two eyes' images. Using adaptive optics, we show that even minute optical aberrations that are not clinically correctable, and go unnoticed in everyday vision, can affect stereo acuity. Hence, the human binocular system is capable of using fine details that are not experienced in everyday vision. Interestingly, stereo acuity varied considerably across individuals even when they were provided identical perfect optics. We also found that individuals' stereo acuity is better when viewing with their habitual optics rather than someone else's (better) optics. Together, these findings suggest that the visual system compensates for habitual optical aberrations through neural adaptation and thereby optimizes stereovision uniquely for each individual. Thus, stereovision is limited by small optical aberrations and by neural adaptation to one's own optics.

The blur horopter: Retinal conjugate surface in binocular viewing.

From measurements of wavefront aberrations in 16 emmetropic eyes, we calculated where objects in the world create best-focused images across the central 27\(^\circ\) (diameter) of the retina. This is the retinal conjugate surface. We calculated how the surface changes as the eye accommodates from near to far and found that it mostly maintains its shape. The conjugate surface is pitched top-back, meaning that the upper visual field is relatively hyperopic compared to the lower field. We extended the measurements of best image quality into the binocular domain by considering how the retinal conjugate surfaces for the two eyes overlap in binocular viewing. We call this binocular extension the blur horopter. We show that in combining the two images with possibly different sharpness, the visual system creates a larger depth of field of apparently sharp images than occurs with monocular viewing. We examined similarities between the blur horopter and its analog in binocular vision: the binocular horopter. We compared these horopters to the statistics of the natural visual environment. The binocular horopter and scene statistics are strikingly similar. The blur horopter and natural statistics are qualitatively, but not quantitatively, similar. Finally, we used the measurements to refine what is commonly referred to as the zone of clear single binocular vision.

Lags and leads of accommodation in humans: Fact or fiction?

The focusing response of the human eye - accommodation - exhibits errors known as lags and leads. Lags occur when the stimulus is near and the eye appears to focus farther than the stimulus. Leads occur with far stimuli where the eye appears to focus nearer than the stimulus. We used objective and subjective measures simultaneously to determine where the eye is best focused. The objective measures were made with a wavefront sensor and an autorefractor, both of which analyze light reflected from the retina. These measures exhibited typical accommodative errors, mostly lags. The subjective measure was visual acuity, which of course depends not only on the eye's optics but also on photoreception and neural processing of the retinal image. The subjective measure revealed much smaller errors. Acuity was maximized at or very close to the distance of the accommodative stimulus. Thus, accommodation is accurate in terms of maximizing visual performance.

Crossed-uncrossed projections from primate retina are adapted to disparities of natural scenes.

In mammals with frontal eyes, optic-nerve fibers from nasal retina project to the contralateral hemisphere of the brain, and fibers from temporal retina project ipsilaterally. The division between crossed and uncrossed projections occurs at or near the vertical meridian. If the division was precise, a problem would arise. Small objects near midline, but nearer or farther than current fixation, would produce signals that travel to opposite hemispheres, making the binocular disparity of those objects difficult to compute. However, in species that have been studied, the division is not precise. Rather, there are overlapping crossed and uncrossed projections such that some fibers from nasal retina project ipsilaterally as well as contralaterally and some from temporal retina project contralaterally as well as ipsilaterally. This increases the probability that signals from an object near vertical midline travel to the same hemisphere, thereby aiding disparity estimation. We investigated whether there is a deficit in binocular vision near the vertical meridian in humans and found no evidence for one. We also investigated the effectiveness of the observed decussation pattern, quantified from anatomical data in monkeys and humans. We used measurements of naturally occurring disparities in humans to determine disparity distributions across the visual field. We then used those distributions to calculate the probability of natural disparities transmitting to the same hemisphere, thereby aiding disparity computation. We found that the pattern of overlapping projections is quite effective. Thus, crossed and uncrossed projections from the retinas are well designed for aiding disparity estimation and stereopsis.

Contributions of foveal and non-foveal retina to the human eye's focusing response.

The human eye changes focus-accommodates-to minimize blur in the retinal image. Previous work has shown that stimulation of nonfoveal retina can produce accommodative responses when no competing stimulus is presented to the fovea. In everyday situations it is very common for the fovea and other parts of the retina to be stimulated simultaneously. We examined this situation by asking how nonfoveal retina contributes to accommodation when the fovea is also stimulated. There were three experimental conditions. (a) Real change in which stimuli of different sizes, centered on the fovea, were presented at different optical distances. Accommodation was, as expected, robust because there was no conflicting stimulation of other parts of the retina. (b) Simulated change, no conflict in which stimuli of different sizes, again centered on the fovea, were presented at different simulated distances using rendered chromatic blur. Accommodation was robust in this condition because there was no conflict between the central and peripheral stimuli. (c) Simulated change, conflict in which a central disk (of different diameters) was presented along with an abutting peripheral annulus. The disk and annulus underwent opposite changes in simulated distance. Here we observed a surprisingly consistent effect of the peripheral annulus. For example, when the diameter of the central stimulus was 8° (thereby stimulating the fovea and parafovea), the abutting peripheral annulus had a significant effect on accommodation. We discuss how these results may help us understand other situations in which nonfixated targets affect the ability to focus on a fixated target. We also discuss potential implications for the development of myopia and for foveated rendering.

Binocular Eye Movements Are Adapted to the Natural Environment.

Humans and many animals make frequent saccades requiring coordinated movements of the eyes. When landing on the new fixation point, the eyes must converge accurately or double images will be perceived. We asked whether the visual system uses statistical regularities in the natural environment to aid eye alignment at the end of saccades. We measured the distribution of naturally occurring disparities in different parts of the visual field. The central tendency of the distributions was crossed (nearer than fixation) in the lower field and uncrossed (farther) in the upper field in male and female participants. It was uncrossed in the left and right fields. We also measured horizontal vergence after completion of vertical, horizontal, and oblique saccades. When the eyes first landed near the eccentric target, vergence was quite consistent with the natural-disparity distribution. For example, when making an upward saccade, the eyes diverged to be aligned with the most probable uncrossed disparity in that part of the visual field. Likewise, when making a downward saccade, the eyes converged to enable alignment with crossed disparity in that part of the field. Our results show that rapid binocular eye movements are adapted to the statistics of the 3D environment, minimizing the need for large corrective vergence movements at the end of saccades. The results are relevant to the debate about whether eye movements are derived from separate saccadic and vergence neural commands that control both eyes or from separate monocular commands that control the eyes independently.SIGNIFICANCE STATEMENT We show that the human visual system incorporates statistical regularities in the visual environment to enable efficient binocular eye movements. We define the oculomotor horopter: the surface of 3D positions to which the eyes initially move when stimulated by eccentric targets. The observed movements maximize the probability of accurate fixation as the eyes move from one position to another. This is the first study to show quantitatively that binocular eye movements conform to 3D scene statistics, thereby enabling efficient processing. The results provide greater insight into the neural mechanisms underlying the planning and execution of saccadic eye movements.

Creating correct blur and its effect on accommodation.

Blur occurs naturally when the eye is focused at one distance and an object is presented at another distance. Computer-graphics engineers and vision scientists often wish to create display images that reproduce such depth-dependent blur, but their methods are incorrect for that purpose. They take into account the scene geometry, pupil size, and focal distances, but do not properly take into account the optical aberrations of the human eye. We developed a method that, by incorporating the viewer's optics, yields displayed images that produce retinal images close to the ones that occur in natural viewing. We concentrated on the effects of defocus, chromatic aberration, astigmatism, and spherical aberration and evaluated their effectiveness by conducting experiments in which we attempted to drive the eye's focusing response (accommodation) through the rendering of these aberrations. We found that accommodation is not driven at all by conventional rendering methods, but that it is driven surprisingly quickly and accurately by our method with defocus and chromatic aberration incorporated. We found some effect of astigmatism but none of spherical aberration. We discuss how the rendering approach can be used in vision science experiments and in the development of ophthalmic/optometric devices and augmented- and virtual-reality displays.

Identifying Computer-Generated Portraits: The Importance of Training and Incentives.

The past two decades have seen remarkable advances in photo-realistic rendering of everything from inanimate objects to landscapes, animals, and humans. We previously showed that despite these tremendous advances, human observers remain fairly good at distinguishing computer-generated from photographic images. Building on these results, we describe a series of follow-up experiments that reveal how to improve observer performance. Of general interest to anyone performing psychophysical studies on Mechanical Turk or similar platforms, we find that observer performance can be significantly improved with the proper incentives.

The natural statistics of blur.

Blur from defocus can be both useful and detrimental for visual perception: It can be useful as a source of depth information and detrimental because it degrades image quality. We examined these aspects of blur by measuring the natural statistics of defocus blur across the visual field. Participants wore an eye-and-scene tracker that measured gaze direction, pupil diameter, and scene distances as they performed everyday tasks. We found that blur magnitude increases with increasing eccentricity. There is a vertical gradient in the distances that generate defocus blur: Blur below the fovea is generally due to scene points nearer than fixation; blur above the fovea is mostly due to points farther than fixation. There is no systematic horizontal gradient. Large blurs are generally caused by points farther rather than nearer than fixation. Consistent with the statistics, participants in a perceptual experiment perceived vertical blur gradients as slanted top-back whereas horizontal gradients were perceived equally as left-back and right-back. The tendency for people to see sharp as near and blurred as far is also consistent with the observed statistics. We calculated how many observations will be perceived as unsharp and found that perceptible blur is rare. Finally, we found that eye shape in ground-dwelling animals conforms to that required to put likely distances in best focus.

Dynamic lens and monovision 3D displays to improve viewer comfort.

Stereoscopic 3D (S3D) displays provide an additional sense of depth compared to non-stereoscopic displays by sending slightly different images to the two eyes. But conventional S3D displays do not reproduce all natural depth cues. In particular, focus cues are incorrect causing mismatches between accommodation and vergence: The eyes must accommodate to the display screen to create sharp retinal images even when binocular disparity drives the eyes to converge to other distances. This mismatch causes visual discomfort and reduces visual performance. We propose and assess two new techniques that are designed to reduce the vergence-accommodation conflict and thereby decrease discomfort and increase visual performance. These techniques are much simpler to implement than previous conflict-reducing techniques. The first proposed technique uses variable-focus lenses between the display and the viewer's eyes. The power of the lenses is yoked to the expected vergence distance thereby reducing the mismatch between vergence and accommodation. The second proposed technique uses a fixed lens in front of one eye and relies on the binocularly fused percept being determined by one eye and then the other, depending on simulated distance. We conducted performance tests and discomfort assessments with both techniques and compared the results to those of a conventional S3D display. The first proposed technique, but not the second, yielded clear improvements in performance and reductions in discomfort. This dynamic-lens technique therefore offers an easily implemented technique for reducing the vergence-accommodation conflict and thereby improving viewer experience.

Stereoscopic depth constancy.

Depth constancy is the ability to perceive a fixed depth interval in the world as constant despite changes in viewing distance and the spatial scale of depth variation. It is well known that the spatial frequency of depth variation has a large effect on threshold. In the first experiment, we determined that the visual system compensates for this differential sensitivity when the change in disparity is suprathreshold, thereby attaining constancy similar to contrast constancy in the luminance domain. In a second experiment, we examined the ability to perceive constant depth when the spatial frequency and viewing distance both changed. To attain constancy in this situation, the visual system has to estimate distance. We investigated this ability when vergence, accommodation and vertical disparity are all presented accurately and therefore provided veridical information about viewing distance. We found that constancy is nearly complete across changes in viewing distance. Depth constancy is most complete when the scale of the depth relief is constant in the world rather than when it is constant in angular units at the retina. These results bear on the efficacy of algorithms for creating stereo content.This article is part of the themed issue 'Vision in our three-dimensional world'.